Adaptive contrast optimization of digital color images

ABSTRACT

Methods and systems are provided for adjusting the contrast and intensity of digital images. Digital images may be processed by first creating a histogram that identifies the intensity of components of the digital image. A transformation function is then computed using the histogram. The transformation function can be used to adjust the intensity of a low light portion of the digital image and is applied to individual color components of the digital image. Finally, the contrast of a portion of the digital image other than the low light portion of the digital image is adjusted.

FIELD OF INVENTION

The invention relates to the processing of digital images. Moreparticularly, aspects of the invention relate to systems and methods foradjusting the contrast and intensity of digital images.

BACKGROUND

It has become common for portable devices, such as mobile terminals, toinclude digital cameras. Mobile terminals equipped with cameras allowusers to capture images and video and then transmit the captured videoand images to remote locations. Because of the size limitations placedon cameras and sensors that are part of mobile terminals, it is commonfor captured images to have poor quality. In particular, contrast andintensity levels are often set at levels that result in poor imagequality.

Scenes that contain both dark and bright sections often result in poorcontrast and intensity levels, particularly when automatic settings areused. The camera exposure time is set for a short time period tominimize the number of saturated pixels. The short exposure time causessome parts of the image to have low intensity and contrast. Shortexposure times may also be set manually to capture moving objectswithout blurring. Images captured with manually set short exposure timesmay have similar intensity and contrast limitations.

There is a need in the art for improved systems and methods forprocessing digital images to correct intensity and/or contrast levels.

SUMMARY

Embodiments of the present invention address one or more of the problemsdescribed above by providing systems and methods for correctingintensity and contrast levels of digital images while providing largedynamic range and minimizing contrast distortion. In one embodiment, anintensity image that identifies the intensity of pixels of an inputimage is first computed. A histogram of the intensity image is thencreated. Next, a transformation function is computed using thehistogram. The transformation function is computed such that it can beused to adjust the intensity of a low light portion of the digitalimage. The transformation function is then applied to individual colorcomponents of the digital image. Finally, the contrast of a portion ofthe digital image other than the low light portion of the digital imageis adjusted.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. The Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary of the invention, as well as the followingdetailed description of illustrative embodiments, is better understoodwhen read in conjunction with the accompanying drawings, which areincluded by way of example, and not by way of limitation with regard tothe claimed invention.

FIG. 1 illustrates an exemplary mobile terminal that may be used toimplement aspects of the present invention.

FIG. 2 illustrates a method of processing digital images in accordancewith an embodiment of the invention.

FIG. 3 illustrates an exemplary histogram that includes intensity dataof an intensity image in accordance with an embodiment of the invention.

FIG. 4 illustrates an exemplary method that may be used to compute atransformation function in accordance with an embodiment of theinvention.

FIG. 5 illustrates an exemplary transformation function in accordancewith an embodiment of the invention.

FIG. 6A illustrates an image prior to undergoing the image processingherein.

FIG. 6B illustrates an image that has undergone processing in accordancewith an embodiment of the invention.

DETAILED DESCRIPTION

In the following description of various illustrative embodiments,reference is made to the accompanying drawings, which form a parthereof, and in which is shown, by way of illustration, variousembodiments in which the invention may be practiced. It is to beunderstood that other embodiments may be utilized and structural andfunctional modifications may be made without departing from the scope ofthe present invention.

Aspects of the invention may be implemented with digital cameras anddevices that include digital cameras. Devices that include digitalcameras include mobile terminals and portable computer devices. FIG. 1illustrates an exemplary mobile terminal 10 that may be used toimplement aspects of the invention. As shown in FIG. 1, a mobileterminal 10 may include a processor 128 connected to a user interface130, a memory 134 and/or other storage, a display 136 and a camera 156.Mobile terminal 10 may also include a battery 150, a speaker 152 andantennas 154. User interface 130 may further include a keypad, touchscreen, voice interface, one or more arrow keys, joy-stick, data glove,mouse, roller ball, touch screen, or the like.

Computer executable instructions and data used by processor 128 andother components within mobile terminal 10 may be stored in computerreadable memory 134. Memory 134 may be implemented with any combinationof read only memory modules or random access memory modules, optionallyincluding both volatile and nonvolatile memory and optionally beingdetachable. Software 140 may be stored within memory 134 and/or storageto provide instructions to processor 128 for enabling mobile terminal 10to perform various functions. Alternatively, some or all of mobileterminal 10 computer executable instructions may be embodied in hardwareor firmware (not shown).

Mobile terminal 10 may be configured to send and receive transmissionsbased on the Bluetooth standard, through a specific Bluetooth module141. Additionally, mobile terminal 10 may also be configured to receive,decode and process transmissions through an FM/AM radio receiver 142, awireless local area network (WLAN) transceiver 143, and atelecommunications transceiver 144. In one aspect of the invention,mobile terminal 10 may receive radio data stream (RDS) messages. Mobileterminal 10 may be equipped with other receivers/transceivers, e.g., oneor more of a Digital Audio Broadcasting (DAB) receiver, a Digital RadioMondiale (DRM) receiver, a Forward Link Only (FLO) receiver, a DigitalMultimedia Broadcasting (DMB) receiver, etc. Hardware may be combined toprovide a single receiver that receives and interprets multiple formatsand transmission standards, as desired. That is, each receiver in amobile terminal device may share parts or subassemblies with one or moreother receivers in the mobile terminal device, or each receiver may bean independent subassembly.

FIG. 2 illustrates a method of processing digital images in accordancewith an embodiment of the invention. First, in step 202 and intensityimage that identifies the intensity of pixels of an input image iscomputed. The input image may be in a compressed or uncompressed RGBformat such as JPEG and TIFF. In some embodiments the input image may bein the form of unprocessed data received from an image sensor of adigital camera. The intensity image may be computed using the followingformula:

$\begin{matrix}{{y\left( {i,j} \right)} = \frac{{r\left( {i,j} \right)} + {b\left( {i,j} \right)} + {g\left( {i,j} \right)}}{3}} & (1)\end{matrix}$

where i identifies a pixel row,j identifies a pixel column. Theintensity of individual red, green and blue pixels are identified byr(i,j), g(i,j) and b(i,j) respectively.

Next, in step 204 a histogram of the intensity image is created. Thetype of histogram created in step 204 may be a conventional histogramused in connection with the processing of typical digital images. FIG. 3illustrates an exemplary histogram 300 that includes intensity data ofan intensity image. As is known in the art, the quantities of low lightpixels are shown toward the left of the histogram and the quantities ofbright pixels are shown toward the right. Histogram 300 corresponds toan image having a large number of low light or dark pixels andrelatively few bright pixels.

In one embodiment, the histogram created in step 204 may be defined asfollows:

h(m)=nr of pixels in the range [m−0.5;m+0.5] with m=0, . . . , 255   (2)

In step 206 a transformation function is computed to adjust theintensity of a low light portion of the digital image. The computationperformed in step 206 may use the histogram created in step 204. FIG. 4illustrates an exemplary method that may be used to compute thetransformation function. First, in step 402 the maximum intensity valueMAX in the low light portion of the histogram and its position M areestimated. In one embodiment:

$\begin{matrix}{{{MAX} = {\max\limits_{{m = 1},\ldots \mspace{11mu},128}\left\{ {h(m)} \right\}}}{and}{M = \left\{ {{m = 1},\ldots \mspace{11mu},{128\left. {{h(m)} = {MAX}} \right\}}} \right.}} & (3)\end{matrix}$

In this manner the maximum number of pixels that are in the low lightportion of the image and the corresponding number of pixels are found.The position M in the histogram corresponds to the value MAX. The lowlight portion of the histogram is analyzed so that the intensity andcontrast of dark regions of the image can be adjusted.

In step 404, the width of the low light portion of the histogram isestimated. Step 404 may be used to determine the portion of the lowlight region containing the most pixels. In one embodiment, a thresholdvalue, such as 0.3, is selected and the range of the histogram[Imin,Imax] is computed as:

$\begin{matrix}{{low} = \left\{ {{m = 1},\ldots \mspace{11mu},{{M\left. {{h(m)} > {0.3\; {MAX}}} \right\} {and}R\; 1} = {\min\limits_{m}\left\{ {low} \right\}}}} \right.} & (4) \\{{high} = \left\{ {{m = M},\ldots \mspace{11mu},{{128\left. {{h(m)} > {0.3\; {MAX}}} \right\} {and}R\; 2} = {\max\limits_{m}\left\{ {high} \right\}}}} \right.} & (5)\end{matrix}$Imin=M−max{|M−R1|,|M−R2|}and Imax=M+max{|M−R1|,|M−R2|}  (6)

Input images that have a large number of dark pixels that have almostthe same intensity level will result in a small range (Imin and Imaxwill be close to each other). The amount of processing that ultimatelytakes place is inversely related to the range. Input images having largeranges already have a large dynamic range and require less processing.In some embodiments the maximum and the minimum strength of theprocessing are limited in order to limit over-processing or smoothingthe input image.

Finally, in step 406 parameters of the transformation function areestimated and the transformation function is fit to the histogram. Thetransformation function may be fitted to the range [Imin,Imax]determined above. In one embodiment, the transformation function f(i,j)is defined as

$\begin{matrix}{{f\left( {i,j} \right)} = \frac{255}{1 + {\exp \left( {- \frac{K\left( {{y\left( {i,j} \right)} - M} \right.}{255}} \right)}}} & (7)\end{matrix}$

A scaling factor of 255 is used to normalize the intensity pixel valuesin the range [−1,1]. FIG. 5 illustrates an exemplary transformationfunction 500. The maximum slope exists for inputs in the range [−1,1]and this is the reason why the pixel values must be scaled. Subtractingthe maximum intensity position M shifts the center of the correctionfunction to the position of the maximum of the histogram.

The value K in equation can result in saturation at the zero level orthe 255 level if not chosen properly. A very large value of K for atransformation function enhance the contrast of the pixels that havevalues close to M but the other pixels will be simply put to 0 or 255.That is all pixels not located close to M will be black or white. K mayhave a non-fixed value when there is a desire for high contrast whileavoiding over saturation. A wide range [Imin,Imax] may result in a smallK that prevents saturation of the other pixels and also decrease thestrength of the processing. A narrow range [Imin,Imax] may result in alarger K in order to better increase the contrast and intensity. Howeverin any case the pixels that are at the extreme value of the intensity(around 0 and around 255 intensity levels) should not be saturated.

In one embodiment, to compute the value of the parameter K we start fromthe following condition imposed on the transformation function f(i,j):

ƒ(Imax)=0.7*255   (8)

This value ensures a remaining dynamic range of 0.3*255 for pixels thatare situated around 0 levels and also around 255 level. Solving equation(8) for K results in the following formula for adaptively selecting thevalue of this parameter:

$\begin{matrix}{K = {\frac{{- 255}\; {\ln \left( \frac{1 - 0.7}{0.7} \right)}}{{Imax} - M} = \frac{{- 255}\; {\ln (0.43)}}{{Imax} - M}}} & (9)\end{matrix}$

The scaling factor 255 ensures that the pixels after applying thecorrection function are on the range [0,255]. In some situations thevalue of the parameter K, estimated by equation (9), can be too large orto small. In such situations its value should be bounded. An exemplaryminimum value Kmin equals 3 and an exemplary maximum value K maxε[8,10]. The fixed value Kmin may be chosen such that the transformationfunction f(i,j) is linear inside the input range [−1,1]. The maximumvalue of K can be selected fixed or it can be a user defined parameter.Modifications to Kmax can increase or decrease the maximum strength ofthe processing.

Returning to FIG. 2, in step 208 the transformation function is appliedto individual color components of the digital image. In oneimplementation, for the red component r(i,j), the minimum and maximumvalues of r(i,j) are determined so that they may be preserved asfollows:

$\begin{matrix}{{{min\_ r} = {\min\limits_{i,j}\left( {r\left( {i,j} \right)} \right)}}{{max\_ r} = {\max\limits_{i,j}\left( {r\left( {i,j} \right)} \right)}}} & (10)\end{matrix}$

Correction may then be performed as follows:

$\begin{matrix}{{\hat{r}\left( {i,j} \right)} = \frac{max\_ r}{1 + {\exp \left( {- \frac{K\left( {{r\left( {i,j} \right)} - M} \right)}{255}} \right)}}} & (11)\end{matrix}$

The minimum and the maximum pixel values of the red color component maybe restored as follows:

$\begin{matrix}{{{\hat{r}\left( {i,j} \right)} = {{\hat{r}\left( {i,j} \right)} - {min\_ r}}}{{\hat{r}\left( {i,j} \right)} = {\frac{{max\_ r} - {min\_ r}}{\max \left( {\hat{r}\left( {i,j} \right)} \right)}{\hat{r}\left( {i,j} \right)}}}{{\hat{r}\left( {i,j} \right)} = {{\hat{r}\left( {i,j} \right)} + {min\_ r}}}} & (12)\end{matrix}$

During the correction phase, some pixels from the high intensity part ofthe input image may be over processed. Over saturation can be correctedby using a weighted average, such as:

{circumflex over (r)}(i,j)=(1=a(i,j)){circumflex over(r)}(i,j)+a(i,j)r(i,j)   (13)

With a(i,j) being a weighting function computed as follows:

$\begin{matrix}{{a\left( {i,j} \right)} = \left\{ \begin{matrix}{0} & {{{{if}\mspace{14mu} {r\left( {i,j} \right)}} < M}} \\{\frac{{f\left( {y\left( {i,j} \right)} \right)} - 0.5}{255}} & {{otherwise}}\end{matrix} \right.} & (14)\end{matrix}$

The processing described above with respect to red may then be repeatedfor other color components, such as green and blue.

Finally, in step 210 the contrast of a portion of the digital imageother than the low light portion of the digital image is adjusted. Inone embodiment, step 210 includes modifying the level of the localcontrast, but not the intensity. The transformation function f(i,j) maybe used to indicate which pixels were processed in the previous steps.

The contrast of a portion of the digital image other than the low lightportion of the digital image may be adjusted by first computing a gainmap g(i,j) according to:

$\begin{matrix}{{{g\left( {i,j} \right)} = \frac{1}{1 + {\exp \left( {- \frac{K\left( {{r\left( {i,j} \right)} - M} \right)}{255}} \right)}}}{{g\left( {i,j} \right)} = {{g\left( {i,j} \right)} - {\min \left( {g\left( {i,j} \right)} \right)}}}{{g\left( {i,j} \right)} = {\frac{g\left( {i,j} \right)}{\max \left( {g\left( {i,j} \right)} \right)} + 1}}} & (14)\end{matrix}$

In one implementation, a maximum value of the gain map is 2 and it isobtained for the pixels that were not processed at previous steps. Ofcourse a maximum value larger than 2 can be used in order to have astronger contrast processing. In an alternative embodiment, the maximumvalue of the gain can also be selected as a user defined parameter whichcan be used to fine tune the processing quality.

Vertical and horizontal sums and differences may be computed as follows:

d1={circumflex over (r)}(i,j)−{circumflex over (r)}(i+1,j)

s1={circumflex over (r)}(i,j)+{circumflex over (r)}(i+1,j)

d2={circumflex over (r)}(i,j)−{circumflex over (r)}(i,j+1)

s1={circumflex over (r)}(i,j)+{circumflex over (r)}(i,j+1)   (15)

Next, the portion of the digital image other than the low light portionof the digital image may be adjusted by multiplication of thedifferences with the gain map as follows:

{circumflex over (d)}1(i,j)=g(i,j)d1(i,j)

{circumflex over (d)}2(i,j)=g(i,j)d2)(i,j)   (16)

The image may then be reconstructed according to:

$\begin{matrix}{{{Rout}\left( {i,j} \right)} = \frac{{\hat{d}\; 1\left( {i,j} \right)} + {\hat{d}\; 2\left( {i,j} \right)} + {s\; 1\left( {i,j} \right)} + {s\; 2\left( {i,j} \right)}}{4}} & (17)\end{matrix}$

Finally, image clipping may be performed according to:

$\begin{matrix}{{{Rout}\left( {i,j} \right)} = \left\{ \begin{matrix}{0} & {{{{if}\mspace{14mu} {{Rout}\left( {i,j} \right)}} < 0}} \\{255} & {{{{if}\mspace{14mu} {{Rout}\left( {i,j} \right)}} > 255}} \\{{{Rout}\left( {i,j} \right)}} & {{otherwise}}\end{matrix} \right.} & (18)\end{matrix}$

The remaining color components may be processed using similarprocedures.

FIG. 6A illustrates an image 600 prior to undergoing the imageprocessing described above. FIG. 6B illustrates an image 602 that hasundergone processing in accordance with an embodiment of the invention.One can see that the intensity of the low light portion of image 602exceeds the intensity of the low light portion of image 600.

While illustrative systems and methods as described herein embodyingvarious aspects of the present invention are shown, it will beunderstood by those skilled in the art, that the invention is notlimited to these embodiments. Modifications may be made by those skilledin the art, particularly in light of the foregoing teachings. Forexample, each of the elements of the embodiments may be utilized aloneor in combination or subcombination with elements of the otherembodiments. It will also be appreciated and understood thatmodifications may be made without departing from the true spirit andscope of the present invention. The description is thus to be regardedas illustrative instead of restrictive on the present invention.

1. A method of processing a digital image, the method comprising: (a) computing an intensity image that identifies the intensity of pixels of an input image; (b) creating a histogram of the intensity image; (c) computing a transformation function, using the histogram, that can be used to adjust the intensity of a low light portion of the digital image; and (d) applying the transformation function to individual color components of the digital image.
 2. The method of claim 1, wherein (a) comprises determining the intensity of red, green and blue pixels.
 3. The method of claim 2, wherein (a) comprises computing an intensity image y(i,j), such that ${y\left( {i,j} \right)} = \frac{{r\left( {i,j} \right)} + {b\left( {i,j} \right)} + {g\left( {i,j} \right)}}{3}$ where i identifies a pixel row and j identifies a pixel column and r(i,j), g(i,j) and b(i,j) are the intensities of red, green and blue color components of the input image.
 4. The method of claim 3, wherein the histogram of the intensity image y(i,j) is defined such that h(m)=nr of pixels in the range [m−0.5;m+0.5]with m=0, . . . ,
 255. 5. The method of claim 1, wherein the individual color components comprise red, green and blue color components.
 6. The method of claim 1, wherein (c) comprises: (i) estimating a maximum number of pixels in a low light portion of the histogram; (ii) estimating a width of the low light portion of the histogram by determining a range that has intensity values that exceed a minimum intensity threshold; and (iii) computing the transformation function to apply to the low light portion of the digital image.
 7. The method of claim 1, wherein (d) further comprises preserving minimum and maximum intensity values of each of the individual color components.
 8. The method of claim 1, further including: (e) adjusting the contrast of a portion of the digital image other than the low light portion of the digital image.
 9. The method of claim 1, wherein the adjustment in (e) is a function of the transformation function.
 10. An apparatus comprising: a sensor that captures digital images; a processor programmed with computer-executable instructions to process a captured digital image by performing the steps comprising: (a) computing an intensity image that identifies the intensity of pixels of the captured digital image; (b) creating a histogram of the intensity image; (c) computing a transformation function, using the histogram, that can be used to adjust the intensity of a low light portion of the captured digital image; and (d) applying the transformation function to individual color components of the captured digital image.
 11. The apparatus of claim 10, wherein the apparatus comprises a digital camera.
 12. The apparatus of claim 10, wherein the apparatus comprises a mobile terminal.
 13. The apparatus of claim 10, wherein (a) comprises computing an intensity image y(i,j), such that ${y\left( {i,j} \right)} = \frac{{r\left( {i,j} \right)} + {b\left( {i,j} \right)} + {g\left( {i,j} \right)}}{3}$ where i identifies a pixel row and j identifies a pixel column and r(i,j), g(i,j) and b(i,j) are the red, green and blue color components of the captured digital image.
 14. The apparatus of claim 10, wherein the histogram of the intensity image y(i,j) is defined such that h(m)=nr of pixels in the range [m−0.5;m+0.5] with m=0, . . . ,
 255. 15. The apparatus of claim 10, wherein (c) comprises: (i) estimating a maximum number of pixels in a low light portion of the histogram; (ii) estimating a width of the low light portion of the histogram by determining a range that has intensity values that exceed a minimum intensity threshold; and (iii) computing the transformation function to apply to the low light portion of the captured digital image.
 16. The apparatus of claim 10, wherein the processor is further programmed with computer-executable instructions to process a captured digital image by performing the step comprising: (e) adjusting the contrast of a portion of the captured digital image other than the low light portion of the captured digital image.
 17. A computer-readable medium containing computer-executable instructions for processing a digital image by performing the steps comprising: (a) creating a histogram that identifies the intensity of components of a digital image; (b) computing a transformation function, using the histogram, that can be used to adjust the intensity of a low light portion of the digital image; (c) applying the transformation function to individual color components of the digital image; and (d) adjusting the contrast of a portion of the digital image other than the low light portion of the digital image.
 18. The computer-readable medium of claim 17, wherein (b) comprises (i) estimating a maximum number of pixels in a low light portion of the histogram; (ii) estimating a width of the low light portion of the histogram by determining a range that has intensity values that exceed a minimum intensity threshold; and (iii) computing the transformation function to apply to the low light portion of the captured digital image.
 19. An apparatus comprising: a means for capturing a digital image; and a means for adjusting intensity of a low light portion of a digital image and contrast of a portion of the digital image other than the low light portion of the digital image.
 20. The apparatus of claim 19, further including: a means for sending and receiving data. 